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EXECUTIVE SUMMARY 
The Afterschool Centers on Education (ACE) is the program administered through the Texas 
Education Agency (TEA) for the federally funded 21* Century Community Learning Center 
(CCLC) grants authorized under Title IV, Part B, of the 2015 Every Student Succeeds Act 
(Public Law 114-95). The Austin Independent School District (AISD) received Cycle 9 218*CCLC 
funding to provide a comprehensive range of out-of-school-time (OST) academic assistance, 
academic enrichment, college and career readiness, and family engagement activities. At the 
local level, the program is known as ACE Austin. Building on the existing infrastructure of 
evidence-based OST activities and partnerships, ACE Austin collaborates with a range of 
partners to provide a comprehensive menu of before-school, afterschool, and summer 
programming to reinforce and complement our regular academic programs. The ACE Austin 
Cycle 9 program exists to provide intentional afterschool program experiences that are high 
quality, are challenging, and inspire all program participants to improve their school outcomes. 
By serving students at Title I schools, ACE Austin strives to achieve the overarching goal of 
narrowing the achievement gap between economically disadvantaged students and students of 
more affluent families. Across activities and centers, the afterschool program focuses on three 
primary objectives: 

e increase academic achievement 

e decrease school-day absences 


e decrease in-school suspensions 


Key Accomplishments 

The ACE Austin Cycle 9 program is aligned with the campus needs assessments and goals 
identified in the campus improvement plans (CIP) of each center. The findings of this report 
were mixed. For example, regular participants (i.e., students who participated for 45 or more 
days) who were targeted for math improvement had a greater average percentage grade change 
than did nonregular participants (i.e., students who participated for fewer than 45 days) at the 
elementary level only. A greater percentage of regular participants than of nonregular 
participants who were targeted for improvement in school-day attendance improved. Almost all 
ACE program staff agreed the ACE program helped students perform better in academic 
outcomes, attendance, behavior, college/career readiness, and social emotional learning. 
Additionally, the majority of the program staff indicated a desire to continue synchronous live 


virtual activities on Zoom and asynchronous anytime activities in the future. 


Many of the program staff selected the following three professional development areas: aligning 
with school-day academic goals, social emotional learning (SEL), and engaging with families. 
Finally, the majority of the teachers at ACE Austin Cycle 9 schools agreed that the ACE program 
benefited the students and/or families. Despite challenges amidst the COVID-19 pandemic, the 
ACE Austin Cycle 9 program clearly remained committed to providing quality programming that 
was accessible, flexible, and supportive of the development of students’ full potential. Table 1 


summarizes the major key accomplishments, based on Texas 21** CCLC ACE component areas. 


Table 1. 
Summary of Key Accomplishments 


Program measure and outcome Result 
Student population served 
Program quality 


Academics 


Math 


Grade average 


© 
© 
Reading © 
© 
© 
© 


School-day attendance 
Discipline 
Family engagement 
Program impact 
ACE staff perceptions © 


Teachers’ perceptions © 

Note. Independent sample étests were conducted to compare program participants on each student outcome (i.e., 
reading and math grades, average grades, and school-day attendance rate) between regular participants and nonregular 
participants. 

© Indicates a positive outcome for the measure 

© Indicates a neutral or no change for the measure 

@ Indicates a negative outcome for the measure 

Indicates limited or no data available 


Areas for Improvement 

The ACE Austin Cycle 9 program staff remained committed to offering quality programming 
amidst the COVID-19 pandemic. As we strive to go back to normalcy, the ACE Austin Cycle 9 
program staff continue to identify opportunities to assist students in maximizing the benefits of 
program participation. While we achieved a positive impact on many student outcomes this year, 
program managers, site coordinators, and program staff should continue to examine best 
practices to recruit and retain students in the program. Working collaboratively with parents, 
school-day campus teachers, and administrators is key to ensuring that students who need 
assistance are identified and are recruited into the program to take advantage of this free-of- 
charge, quality afterschool programming. Professional development opportunities to improve 


program effectiveness should be provided to ACE staff in areas deemed important. 
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INTRODUCTION AND PURPOSE OF THE PROGRAM 

The Texas Afterschool Centers on Education (ACE) is funded by the 21%t Century Community 
Learning Center (CCLC) federal grant, and is authorized under Title IV, Part B, of the 2015 Every 
Student Succeeds Act (Public Law 114-95). The Texas ACE program is administered through the 
Texas Education Agency (TEA). The Austin Independent School District (AISD) received Cycle 9 
21**CCLC funding to provide a comprehensive range of out-of-school-time (OST) academic 
assistance, academic enrichment, college and career readiness, and family engagement 
activities. At the local level, the program is known as ACE Austin. Building on the existing 
infrastructure of evidence-based OST activities and partnerships, ACE Austin collaborates with a 
range of partners to provide a comprehensive menu of before-school, afterschool, and summer 
programming to complement our regular academic programs. The ACE Austin Cycle 9 program 
exists to provide intentional school program experiences that are high quality, are challenging, 


and inspire all program participants to improve their school outcomes. 


By serving students at Title I schools, ACE Austin strives to achieve the overarching goal of 
narrowing the achievement gap between economically disadvantaged students and students of 
more affluent families. Across activities and centers, the afterschool program focuses on three 
primary objectives: 

e increase academic achievement 

e decrease school-day absences 


e decrease in-school suspensions 


This Texas ACE program is at 32 schools across the district. This report examines outcomes for 
the 2,052 ACE Austin Cycle 9 participants at 10 AISD campuses during the 2020-2021 school 
year: six elementary schools (Langford, Oak Springs, Rodriguez, T.A. Brown, Widen, and 
Wooten) and four middle schools (Bedichek, Dobie, Martin, and Mendez). At each school, 
activities are offered at least 15 hours per week for 31 weeks during the academic year and 16 
hours per week for 6 weeks during the summer. All activities are in one or more of the four 21st 
CCLC core component areas: academic assistance, enrichment, family engagement, and college 


and career readiness (Figure 1). 


Figure 1. 


ACE Austin TX 21* CCLC Core Component Areas 


Family engagement 

ACE Austin staff partner with the AISD 
Adult Education Department and parent 
support specialists to provide family 
engagement activities that help connect 
families to schools and enable them to 
support their students’ academic 
achievement. 


College and career readiness 


The ACE Austin participants are provided 
with activities to help them prepare for 
college and career. Students investigate 
careers, visit area colleges and 
universities, practice public speaking 
skills, and participate in service projects. 


EVALUATION STRATEGY 


Expectations 


Academic assistance 


ACE Austin offers activities designed to 
improve students’ achievement by 
providing extra assistance and support 
through tutoring and homework help for 
students who are struggling in core 
subjects, including science, math, 
reading, and social studies. 


Enrichment 

ACE Austin offers skill-building enrichment 
activities to which some students would 
otherwise lack access, including fine arts, 
technology, games, health and fitness, 
outdoor and environmental education, and 
youth leadership and development. 


The Department of Research and Evaluation (DRE) staff and ACE Austin program staff together 
reviewed the grant requirements and developed an evaluation plan and timeline for the program, 
which were published online, as part of the DRE work plan 
(https://www.austinisd.org/sites/default/files/dept/dre/docs/2021 DRE Evaluation Plans 2020- 
2021.pdf). Throughout the duration of the grant program, evaluators worked closely with 
program staff to collect and submit identified data in a timely fashion and met regularly to 


monitor progress and make any needed adjustments. 


The evaluation plan was used to ensure continuous improvement for (a) program management, 
by monitoring program operation; (b) staying on track, by ensuring the program stayed focused 
on the goals, objectives, strategies, and outcomes; (c) efficiency, by streamlining service delivery 
and lowering the cost of services; (d) accountability, by producing evidence of program effects; 


and (e) sustainability, by providing evidence of effectiveness to all stakeholders. 


The ACE Austin program staff used the TX21*t student tracking system to track students’ program 
attendance and other program data needed for TEA reports. The DRE evaluator extracted students’ 
records from AISD’s data warehouse and assisted program staff with formatting and data entry 


into the TX21* student tracking system to ensure accurate reporting to TEA. 


Measurement 

Program participation files and AISD student records provided demographic information and 
input for each of the school-related outcomes. Despite the challenges of the COVID-19 
pandemic, AISD kept its school buildings and facilities open at a limited capacity, following the 
national safety and state health precautionary guidelines and measures. A wide-array of learning 
models and activities (i.e., in-person, synchronous “live” virtual learning, asynchronous 
“anytime” virtual learning activities, and a hybrid model that combined two or more learning 
models) was offered to students and families to suit their individual needs for optimum learning 


and campus involvement, with safety and health at the core of every programming decision. 


Similar to last year (dela Cruz, Andrews, and Christian, 2020), this year’s end-of-year outcome 
measures were limited to the available data. Due to the COVID-19 pandemic, the usual program 
surveys given to parents and students were not administered this school year, so they could 
focus on matters deemed important and essential related to students’ learning and family life. 
Due to delays in test data availability from TEA at the time this report was written, State of Texas 
Assessments of Academic Readiness (STAAR) could not be used as an outcome variable this year. 
No targeting for average grades was done due to inconsistencies in the reporting periods in the 
prior year. In addition, very little campus discipline occurred, likely due to the majority of 
students attending virtually for much of the school year. This made using campus discipline as 
an outcome not possible. However, the ACE Austin Staff Survey was administered to gather 
feedback about staffs’ experiences with the afterschool program. Additionally, data generated 
from the district-wide surveys, such as the AISD Family Survey and the End-of-Year Multi- 
Program Survey (formerly Employee Coordinated Survey), that were relevant to ACE 


programming were analyzed to support the grant’s reporting requirements. 


Data analyses were conducted to examine the impact of program participation on students’ 
outcomes (i.e., academic achievement in reading and math, average grades across all core 
subjects [English language arts, math, science, and social studies], and school-day attendance), 
based on participation status. Program participation status was categorized based on the total 
number of days students participated in the afterschool program during the 2020-2021 school 
year: regular participants were students who participated for 45 or more days, and nonregular 
participants were students who participated for fewer than 45 days. In this report, independent 
sample t-tests were used to compare regular and nonregular targeted students on academic 


achievement in reading and math, grade average, and school-day attendance. 


Prior to running the appropriate data analyses, descriptive statistics were compiled to report on 


the percentages of targeted participants who showed improvement on each of the outcomes. 


Table 2 presents a summary of the methodology used in this report, based on program 


objectives. 
Table 2. 
Summary of Program Methodology 
Program objective Data analysis Data collection/ source 
Improve participants’ An independent ¢test was used to compare regular ab 
: ; ae ; Program participation 
academic performance in and nonregular targeted program participants : 
, ‘ . file; AISD student grades 
reading and math average grade change in reading and math. 


An independent ¢test was used to compare regular 
Improve grade average = and nonregular program participants’ grade 
average. 
An independent étest was used to compare regular 
and nonregular targeted program participants who — Program participation 
had a school-day attendance rate at or below 90% file; AISD student 
in the prior year and demonstrated an improved attendance 
attendance in the current school year. 


Program participation 
file; AISD student grades 


Improve participants’ 
school-day attendance 


Due to a very small percentage of students who 
experienced in-school suspensions, this outcome is _— Data not available 
not reported. 


Improve participants’ 
discipline 


PROGRAM TIMELINE 

The ACE Austin calendar was full, even during the COVID-19 pandemic. Prior to the opening of 
school year 2020-2021, DRE staff and ACE Austin program staff together reviewed the grant 
requirements and developed an evaluation plan and timeline for the program (Figure 2). 
Equipping program staff with knowledge and skills on afterschool program management and 
implementation (e.g., needs assessments, recruiting and retaining participants, program 
monitoring and quality observations) through continuing professional development is key to 
quality afterschool programming. DRE provided a dashboard to monitor individual student 


progress for targeted students. 


Figure 2. 
ACE Austin Program Timeline 


ACE Austin During School Closures Due to the COVID-19 Pandemic 


Program Planning 
ACE Austin program 
Staff and DRE staff 
collaborated to develop 
the annual evaluation 
plan, formulate 
evaluation question, 
and program timeline, 
based on grant 
requirements and 


Conduct Training on 
Needs Assessments 


Trainings on needs 
assessments, SMART 
goals identification, and 
log model design were 
conducted to ensure 
program staff adhere to 
program design, based 
on prior data. 


Recruiting and 
Retaining Participants 
An open enrolment was 
used to recruit and retain 
Students and to 
supplement campus efforts 
10 support atademic needs, 
while also providing for the 
physical, sodal, and 
otional health of families 


ACE Austin Moving Forward... 


Program Monitoring and 
Quality Observations 
Program staff and teachers 
collaboratively implemented 
re-designed activities for 
Students and families. 
Program monitoring and 
quality observations were 
regularly conducted to ensure; 
program fidelity and 


Summer Program 


ACE summer 
programming was offered 
to equip students with 
knowledge and skills 
needed to prepare them 
for the next stage of thei 
academic and life pursuits 
that complement ther 
regular campus program 


curricular alignment through 
continuous feedback. 


program objectives. 


Introduction of some in-person activities 


Oo. Oo 


Dec. 2020 ~ May 2021 June - july 2021 


oO 


June — july. 2020 


Aug. - Sep. 2020 Oct. 2020 ~ Nov. 2021 


Goals that were specific, measurable, attainable, relevant, and time bound (SMART) were 
identified, and logic models were created for each campus. Additionally, DRE staff helped 
program staff to create an online registration form and interactive dashboards with up-to-date 
student-level data for effective recruitment and retention of program participants. For school 
year 2020-2021, staff determined that open enrollment was needed to optimally recruit and 
retain students and to supplement campus efforts to support students’ academic needs, while 
also providing for the physical, social, and emotional health of families and students. Guided by 
the continuous quality improvement cycle, program monitoring and quality observations were 
regularly conducted to ensure the quality of program implementation and curricular alignment 
through continuous feedback. ACE Austin offered a 6-week summer program to equip students 
with knowledge and skills, based on students’ needs and campus assets. It is hoped that 
providing additional learning time through the ACE Austin summer program would enable 
students to cushion the potential cumulative effect of learning loss due to school building 
closures brought on by the COVID-19 pandemic and would prepare them for the next stage of 


their academic and life pursuits. 


ACE Austin Cycle 9 in the Time of COVID-19 Pandemic 

Since the first stay-at-home orders were issued in mid-March 2020, individuals across the state 
of Texas and the nation have experienced the negative impact of the COVID-19 pandemic on 
their lives and livelihoods. As states enacted safety measures to combat this pandemic, families 
struggled with school building closures, job losses, food insecurity, and more. ACE Austin Cycle 9 
program staff joined local and district efforts to address the urgent needs of children and 


families, while at the same time were dealing with their own uncertain future. 


In response to this crisis, the ACE Austin Cycle 9 program immediately switched gears and 
innovatively redesigned its afterschool programming to stay connected with students and their 
families and to keep them safe, healthy, and engaged in learning. Due to continuously evolving 
health and safety standards developed in response to the COVID-19 pandemic, the reopening of 
schools—and consequently the ACE Austin Cycle 9 program—necessitated a radical rethinking of 
the way the program serves its students and families to meet their unique needs and 
circumstances. This required a collaborative approach that leans into community partnerships 


and builds on the campus’s strengths and assets. 


The program areas remained the same. ACE Austin Cycle 9 continued to offer targeted academic 
support, a variety of enrichment opportunities, and family engagement. Centers were also 
encouraged to intentionally integrate social-emotional supports, both through activities 
designed for social emotional learning (SEL) and by embedding SEL skills into other areas. To 
support the program throughout the COVID-19 pandemic, ACE Austin staff and partners 
adapted services and activities that normally are conducted in-person, to meet the hybrid model. 


These program services and activities included (Figure 3): 


e Asynchronous anytime activities. ACE Austin contracted with Bliss Kid Yoga, All 
Rhythms, Samurai School, Dancin’ Jazzi, 4-H Capital, Variety of Texas, Campfire, 
and Creative Action to provide asynchronous activities. New prerecorded videos 
were posted weekly on the ACE Austin Google site; each contained 45 to 60 minutes 
of content for students to complete at the time that worked best for them. 
Participation was tracked, and students were encouraged to submit photos or videos 
to the site coordinator after they completed each activity. This format was used for 


both student- and family/parent-focused activities. 


Figure 3. 


Program Services and Activities ACE Austin Offered in the Time of COVID-19 


Asynchronous Anytime Activities 
ACE Austin contracted with a variety 
of vendors to provide asynchronous 
activities using pre-recorded videos 

posted weekly. 


Synchronous Live Activities 
Each ACE Austin centered offered synchronous 
7 activities via Zoom, based on student and 


family needs as well as campus assets of each 
campus. 


At-Home Kits 


Centered offered activity kits for families to 
pick up and complete at home. This allowed 
students learning from home an opportunity 
to reintegrate in ACE Austin without being 


on screen. 


In-person Activities 

ACE Austin collaborated with campus 
principals to re-integrate in-person activities 
in alignment with the needs and assets of the 
campus. 


Synchronous live activities. Each ACE Austin center offered synchronous activities 
via Zoom. These activities were specific to the needs of each campus. Activities 
could be led by AISD teachers and teaching assistants, paid vendors, or volunteer-led 
community organizations. This format was used for both student- and parent- 
focused activities. 

At-home kits. Centers offered kits for families to pick up and complete at home. This 
allowed students learning from home an opportunity to participate in ACE Austin 
without being on screen. The contents of the kits were specific to the needs and 
assets of each center. This format was used for both student- and family/parent- 
focused activities. 

In-person activities. ACE Austin worked with campus principals to reintegrate in- 
person activities in alignment with the needs and assets of the campus. In-person 
activities were led only by AISD teachers and staff and were only offered to students, 


not parents. 


Building on strong strategic alliances and tapping campuses’ assets, ACE Austin coordinated 


with AISD Food Service, Family Resource Centers, and other district department and programs to 


supplement the campuses’ efforts to provide for the physical, social, and emotional health of 


families and students. Clearly, the ACE Austin program offered a variety of services and activities 


that were designed to support students and families who experienced a negative impact from the 


COVID-19 pandemic. 


GRANTEE AND CENTER OVERVIEW 


During the 2020-2021 school year, ACE Austin Cycle 9 provided afterschool services at 10 AISD 


campuses: six elementary schools (Langford, Oak Springs, Rodriguez, T.A. Brown, Widen, and 


Wooten) and four middle schools (Bedichek, Dobie, Martin, and Mendez). District data indicated 


that the percentage of students at Cycle 9 campuses who had low socioeconomic status (SES; i.e., 


qualified to receive free or reduced-price lunch) was above district and state averages. Also, the 


percentage of students who were classified as emerging bilingual and the percentage of students 


who were considered at risk of dropping out of school were at least or above district and state 


averages at nine of the ten ACE Austin Cycle 9 schools (Table 3). 


Table 3. 


Cycle 9 Campuses Served and Relevant Demographics 


School 


Langford Elementary School (17 = 526) 
Oak Springs Elementary School (7 = 229) 
Rodriguez Elementary School (7 = 471) 
T. A. Brown Elementary School (7 = 292) 
Widen Elementary School (77 = 446) 
Wooten Elementary School (77 = 457) 
Bedichek Middle School (7 =849) 

Dobie Middle School (77 = 665) 

Martin Middle School (7 = 540) 

Mendez Middle School (7 = 616) 

AISD 

State 


Percentage low 
SES 
95% 
97% 
98% 
97% 
97% 
93% 
87% 
97% 
93% 
96% 
53% 
60% 


Percentage 
emerging bilingual 
57% 
16% 
54% 
72% 
49% 
78% 
31% 
61% 
33% 
56% 
28% 
20% 


Source. 2020-2021 AISD student data; 2019-2020 7exas Academic Performance Report 


Percentage at risk 


75% 
48% 
75% 
82% 
69% 
85% 
70% 
81% 
81% 
89% 
50% 
48% 


Participants 

The ACE Austin Cycle 9 program served 2,052 students and hosted events or activities for 514 
families. Program participants were categorized as regular (i.e., attended the afterschool 
program for 45 days or more) and nonregular participants (i.e., attended the afterschool program 
for fewer than 45 days). Program participants represented over a quarter of the students enrolled 
at Cycle 9 campuses. Almost half (n = 1,025) of the ACE Austin Cycle 9 program participants were 
regular participants. Participation at the secondary schools was less consistent, with greater 
percentages of nonregular participants than of regular participants (Figure 4). Campus-level 


demographics mirrored the cycle-level demographics at 10 campuses (Appendix A). 


Figure 4. 
At the campuses served, 4% to 53% of the student body comprised ACE regular participants. 


Langford ES (7 = 558) 
Oak Springs ES (/7 = 251) EEL 1% 46% 
Rodriguez ES (/7 = 420) 10% 75% 16% 
TA. Brown ES (= 525) 
Widen ES (7 = 393) 9% 30% 62% 
Wooten ES (/7 = 428) 31% 2% 67% 
Bedichek MS (7 = 849) [UA Ey 59% 
Dobie MS (/7 = 739) 13% 13% 74% 
Martin MS (/7= 559) 20% 0% 80% 
Mendez MS (7=649) | Ti 12% 81% 
Overall (/V/ = 5,371) 19% 19% yA 
m Regular participant @ Nonregular participant m Nonparticipant 


Source. TX21* student tracking system 2020-2021; AISD student records 


PROGRAM QUALITY IMPLEMENTATION 

Guided by the ACE Austin continuous quality improvement cycle, programming was developed 
based on the needs of each campus (Figure 5). Before implementation, the project directors met 
with each site coordinator to set goals in the following areas: program operations, 


communication, curriculum alignment, quality of instruction, and program evaluation. 


Individual goals were reviewed mid-year, and adjustments were made. The project directors and 
site coordinator used the ACE Austin Quality Observation Checklist, which was adapted from the 
Youth Program Quality Assessment (YPQA) tool (Smith et al., 2016), to document program- 
quality observations. Recommendations for improvement were received by the site coordinator, 
who then met with the OST instructors. Observers looked for compliance in operational 
functions, program quality, and procedures. In addition, observers checked for fidelity to the 
project plan. Project plans included activity alignment, SMART goals, staff-to-student ratios, 


and student engagement strategies. 


Figure 5. 
ACE Austin Continuous Quality Improvement Cycle 


1. Center-level needs assessment 


5. Reassess and improve Site coordinators, campus leaders, and the 


Program Director collaborate with each 
other to assess the needs of individual 
centers. 


Program outcomes are reassessed and 
needed changes are made for 
continuous quality improvement. 


2. Logic model development 


Following campus needs assessments, logic 
models are designed to guide quality 


4. Quality observations implementation at each center. 


Quality program observations are regularly 
conducted to ensure quality program 
outcomes and fidelity to program 
implementation. 


3. Implementation 


Using logic model as a guide, quality 
implementation is closely monitored and 
programmatic changes are documented 
over time. 


Following campus needs assessments, logic models were designed to guide quality 
implementation at each center. Site coordinators, in collaboration with the project director, 
developed the logic models, which also served as a tool for documenting programmatic changes 
over time. Each center logic model included six components: resources, implementation 


practices, outputs/activities, outputs/participation, intermediate outcomes, and impact. 


Program Quality Observations 

ACE Austin Cycle 9 learning activities were designed to meet the students’ unique needs and 
campuses’ assets. These learning activities included in-person, synchronous live activities, and 
asynchronous anytime activities. For school year 2020-2021, fifty-four observations (2,211 
total minutes) were conducted by the program director and site coordinators at ACE Austin 


Cycle 9 campuses: 70% were synchronous live activities, 30% were asynchronous anytime 


10 


activities, and 19% were in-person activities (Figure 6). Both in-person and synchronous 
activities were observed in real time, while asynchronous activities were observed at any given 


time. 


Figure 6. 
The majority of the ACE Austin Cycle 9 learning activities observed were synchronous live activities. 


Asynchronous, 30% 


Synchronous, 70% 


In-person, 19% 


Source. 2020-2021 ACE Austin program quality observation checklist, = 54 learning activities observed. 


The observers used an electronic rating form, based on the YPQA, that covered seven program 
quality domains: physical safety, emotional safety, clear expectations, introduction, intentional 
skill-building activity, reflection, and choices and voices. The average global rating for each 
domain of the program’s quality observations was assessed using a 5-point Likert scale, but 
framing for each program delivery format (i.e., in-person, synchronous live virtual, and 
asynchronous anytime virtual) was different because the preceding scaled items differed 
depending on the format (see Appendix B). Average global ratings that are closer to 5 indicate 
higher program quality observations for that domain. Note that the “choice and voice” domain 
was not rated for asynchronous anytime activities because the items in that domain were not 


relevant for the anytime format. 


Overall, the ACE Austin Cycle 9 program quality was rated very highly, regardless of program 
format (Figure 7). On average, all domains in all formats had a global average quality rating of 


greater than or equal to 4.6. 
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Figure 7. 
The average global ratings for all program quality domains were very high, regardless of program 


delivery format. 


4.9 4.8 4.8 4.8 4.8 4.8 4.8 4.9 
| | | | || 
Physical safety Emotional safetyChoice and voice Introduction Clear Intentional skill- Reflection 
expectations  buiding activity 


Mln-person Synchronous virtual m Anytime virtual 


Source. 2020-2021 ACE Austin program quality observation checklist 


OUTCOMES 

Across activities and centers, the ACE Austin Cycle 9 primary program goals were to improve 
academic achievement, increase school-day attendance, and improve discipline. Guided by these 
goals, site coordinators examined prior data and targeted students to provide them with a wide 
range of programming activities to improve outcomes. Because we expected the program would 
have a greater positive impact on students who participated more, we compared regular and 
nonregular program participants who were targeted for improvement on two of the student 
outcomes, i.e., academic achievement in reading and math and school-day attendance for school 
year 2020-2021. Targeting was not available for grade averages. Specific data analytic procedures 
and statistical tests, including targeting of program participants for improvement for each 


identified program outcome, are discussed in the following sections. 
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Academic Achievement Outcomes: Grades in Reading and Math 
One of the ACE Austin program goals was to have a positive impact on reading and math 
achievement, based on program participation. Site coordinators focused recruitment efforts and 


offered programming activities to help students who needed assistance in those subject areas. 


Prior to data analyses, ACE Austin Cycle 9 program participants were grouped by program 
participation status and grade-level categories to analyze academic achievement in reading and 
math. Grade-level categories were grouped as follows: early education included participants in 
prekindergarten through grade 2, elementary included participants in grades 3 through 5, and 
secondary included participants in grades 6 through 12. The change in grade average in both 
reading and math was calculated using students’ first grading period grade average and their last 
grading period grade average for school year 2020-2021. Students whose average grades in 
reading or math fell below the campus grade-level average for reading or math during the first 
grading period of the school year were targeted for improvement. Independent sample t-tests 
were conducted to compare the grade average change between the first and last grading period 


for regular and nonregular targeted participants in reading and math. 


Around half of all students improved their grades in reading over the course of the year, 
regardless of grade-level group or participation status (Figure 8). The remaining half maintained 
or did worse on their reading grades over time. It is therefore not surprising that, on average, 
there was no statistically significant difference in the average percentage grade change in 
reading between regular and nonregular targeted participants across grade-level groups. 
However, more regular participants than nonregular participants improved grades in math 
(Figure 9). In fact, regular participants who were targeted for improvement in math showed a 
greater average percentage grade change than did nonregular participants at the elementary 
level. This finding was not only statistically significant, but also was practical and meaningful 


(i.e., effect size of .53). 
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Figure 8. 
Across grade-level groups, around half of the students improved their grades in reading, regardless 
of their participation status. 


Early education Reading 
Regular (7 = 125) 38% 21% 42% 
Nonregular (77 = 193) 16% 40% 44% 
Elementary 
Regular (/7 = 89) 43% 6% 52% 
Nonregular (7 = 206) 37% 10% EL 
Secondary 
Regular (7 = 115) 36% 10% 54% 
Nonregular (/7 = 222) EEL 8% 59% 


m Declined mNochange m Improved 


Source. X21 student tracking system 2020-2021; AISD student records 


Figure 9. 
More regular participants than nonregular participants improved grades in math at the elementary level. 


Early education Math 


Regular (/7 = 122) 20% 44% 36% 
Nonregular (/7 = 160) 56% 34% 


Elementary * 


10% 
Regular (7 = 107) 12% 6% 62% 
Nonregular (/7 = 197) 27% 26% 47% 
3 


Secondary 


y) 
Regular (7 = 120) 3% 9% 58% 
Nonregular (/7 = 225) 36% 9% 55% 


m Declined mNochange Improved 


Source. TX21* student tracking system 2020-2021; AISD student records 


Academic Achievement Outcome: Grade Average 

Another program goal was to improve students’ grade averages. We expected that regular 
program participants would have greater grade averages than those of nonregular participants 
during the school year. In this report, only the core courses (i.e., English language arts, math, 
science, and social studies) were considered in computing the grade averages. Due to 
inconsistencies in the reporting periods from the prior school year, no targeting of students in 


their grade averages was done. 


An independent sample t-test was used to compare participants’ grade averages, based on 
program participation (i.e., regular participants and nonregular participants). Results 
revealed that the difference in grade average between regular and nonregular participants 


was not significant (Figure 10). 
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Figure 10. 
The average grades of regular and nonregular participants were not significantly different. 


Nonreguar (0= 1,267) rer ey 
Regular (7 = 746 


Source. TX21* student tracking system 2020-2021; AISD student records 
All grades were converted to numeric grades (4-point scale) and were coded as follows: 100 = 4.00, 99 = 3.9, 
98 = 3.8... through 70 = 1.0 and 69 <= 0 


Student School-Day Attendance 

Another program goal was to improve school-day attendance. We expected that ACE Austin 
Cycle 9 regular participants would show improvement over last year in their school-day 
attendance rate during 2020-2021. Site coordinators looked at the prior year’s attendance data 
for all students in their schools. Students who had a school-day attendance rate at or below 90% 
in the prior year were targeted for improvement by site coordinators. The impact of program 
participation on the school-day attendance of targeted participants was compared using an 
independent sample t-test. Results showed that of the targeted students, a greater percentage of 
regular participants than nonregular participants improved (Figure 11). This finding is not only 


statistically significant, but also was practical and meaningful (i.e., effect size of .30). 


Figure 11. 
A greater percentage of regular participants than nonregular participants who were targeted for 


improvement in school-day attendance improved. 


Nonregular (/ = 128) 


Regular (/7 = 57) 


m Declined m Improved 


Source. TX21* student tracking system 2020-2021; AISD student records 
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Student Discipline 

Discipline was also examined as one of the program goals. We expected students who regularly 
attended the program would show improvement in their discipline. Since very few students 
attended school in-person across the district, there was very little discipline. In fact, very few 
students (n = 19) of the 5,371 students enrolled at the 10 campuses served by the ACE Austin 
Cycle 9 experienced any PEIMS reportable discipline (e.g., in-school suspensions, out-of-school 
suspensions, removal to disciplinary school, etc.) during this school year. Therefore, there was 


not enough variance in discipline to examine this as an outcome this year. 
Family Engagement 


This school year, the ACE Austin Cycle 9 family engagement specialist served 514 families for 
135 days throughout the 10 campuses. Families participated in at least one or more of the 180 
meaningful family activities that included events (e.g., gardening class, Earth Day reuse class), 
clubs (e.g., Brighter Bites, Time Together), and adult education (e.g., English Second Language 
Maestro En Casas, fitness class). These family engagement activities were designed to provide 
opportunities for families to connect with their students’ schools, spend quality time with their 
students, learn new skills, improve their students’ social emotional skills, and support their 


students’ education and life skills. 


To gather feedback and insights from families and children who attended the family activities 
regarding their experiences of the program, participants were asked to complete a brief survey 
aimed at understanding family engagement. Only 14 participants responded to the survey from 
two campuses served by ACE Austin Cycle 9. Results are not reported; due to the very low 


response rate, results are not representative. 


In addition, both the ACE Student Survey and the ACE Parent Survey were not administered to 
allow students and families with the opportunity to focus their attention on other priorities, 
needs, and issues deemed important during the COVID-19 pandemic. However, the district-wide 


Family Survey included an item that was relevant to ACE Austin programming. 


The purpose of the AISD Family Survey is to gather information about parents’ and guardians’ 
attitudes on various topics at the campus where their child is enrolled. This survey is 
administered annually each spring and is offered in multiple languages. For this year, the overall 
response rate was 33% (district N = 16,333). At ACE Austin Cycle 9 campuses, the majority (91%) 
of the parents and guardians who responded (n = 692) agreed that their child’s school provided 
adequate opportunities to participate in activities and programs after and before school (Figure 
12). 


17 


Figure 12. 
The majority of parents and guardians agreed that their child’s school provided adequate 
opportunities to participate in activities and programs after and before school. 


@ Strongly agree © Agree Disagree mi Strongly disagree 


Source. A\SD Family Survey, 2020-2021 


STAFF'S PERSPECTIVES ON PROGRAM IMPACT 

The ACE Staff Survey was administered to staff to gather information and feedback about their 
program experiences. From ACE Austin Cycle 9, 66 staff responded to the ACE Staff Survey. ACE 
staff agreed that the ACE program helped students perform better in attendance (100%), 
academic outcomes (98%), behavior (98%), college/career readiness (97%), and SEL (98%) (Figure 
13). The majority of the program staff indicated a desire to continue synchronous live activities 
on Zoom (59%) and asynchronous anytime activities in BLEND (73%), should the program return 


to 100% in-person attendance in the future (Figure 14). 


Figure 13. 
ACE program staff at ACE Austin Cycle 9 schools agreed that the program helped students perform 


better in academic outcomes, attendance, behavior, college/career readiness, and SEL. 
Attendance 
Academic outcomes 

Behavior 


Social emotional learning 


io () () 
Ss ZS Ss 
oO oS oS 


College/career readiness 


Source. ACE Staff Survey, 2020-2021 
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Figure 14. 
The majority of program staff indicated a desire to continue synchronous live virtual 
activities and asynchrnous anytime activities in the future. 


Synchronous live activities 
Asynchronous anytime activities 


mYes mNo 


Source. ACE Staff Survey, 2020-2021 


Recognizing the importance of professional development opportunities for program 
effectiveness and improvement, ACE staff were also asked about the areas for professional 
development opportunities. The top three areas of need that program staff reported were: 
aligning with school-day academic goals (41%), social emotional learning (41%), and engaging 


with families (41%) (Figure 15). 


Figure 15. 


ACE program staff indicated a high desire for professional development opportunities in: aligning 
with school-day academic goals, social emotional learning, and engaging with families. 


Percentage of ACE staff who selected each professional development area... 
Aligning with school-day academic goals 
Social emotional learning 


Engaging with families 


6866 


Leadership opportunities ——_§® 
Lesson planning ——@®@ 


Grant requirements (e.g., required number of students and parents, —_—@ 
reporting activity components) 
No professional development needed ——_D 


Understanding observation feedback from ACE Austin Quality Checklist) QD 


@ 


Program expectations (e.g., policies and procedures) 


Other —® 


Source: End-of-Year Multi-Program Survey, 2020-2021 
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AISD is committed to understanding staff members’ feelings and perceptions about various 
work-related topics and district-wide programming aimed at improving employees’ well-being 
and program effectiveness. For this reason, DRE administered the End-of-Year Multi-Program 
Survey was administered in Spring 2021. One of the programming areas for which district 
employees were asked to provide feedback was the ACE program. Specifically, teachers (39 out of 
103, 38% response rate) at ACE Austin Cycle 9 campuses were asked about the benefits of the 
ACE program to students and families. The majority of the teachers reported that the ACE 
program benefited the students and/or families in many ways, such as building positive behavior 
(97%), developing SEL skills (97%), making connections to the school (95%), and mitigating 
learning loss caused by the COVID-19 pandemic (95%) (Figure 16). 


Figure 16. 
The majority of the teachers at ACE Austin Cycle 9 school agreed that the ACE program benefited 
students and families. 

Percentage of teachers who agreed that the ACE program benefited... 


Students build positive behavior 
Students develop their social emotional learning skills ————————_ 
Student and families make connections to the school tM 


pandemic 
Lessons they do in the classroom 


Students are motivated their for school-day attendance 


@ 
® 
S 
Student mitigate learning loss caused by the COVID-19 
@ 
@ 
20% 


Student improve their academic skills in core subjects 


Students turn in their homework regularly ——_—@ 


Source. End-of-Year Multi-program Survey, 2020-2021 
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ACE AUSTIN CHALLENGES AND OPPORTUNITIES 

Reopening campuses during the COVID-19 pandemic was unquestionably challenging in many 
ways. It was particularly challenging to implement and sustain the ACE Austin quality program 
that capitalized on whole child development, based on the unique academic and social- 
emotional needs of students. Challenges, however, also opened the door for some opportunities 
to refine and change program processes and structures to strengthen the ACE Austin quality 


programing and to creatively leverage campus and community assets. 


1. Challenge: ACE Austin program participation was a major concern 
this school year. Some campuses struggled to recruit and retain 
participants. 


When school started operating 100% virtually, some campuses struggled with students’ 
attendance and retention during the school day. This translated into program participation 
issues during out-of-school time. Students were no longer interested in spending additional time 


online. 


Opportunity: ACE Austin staff offered numerous participation 
opportunities and used accessible and user-friendly systems to recruit 
and retain participants. 


Taking advantage of the flexibility of program recruitment, ACE Austin program staff 
innovatively offered numerous participation opportunities (synchronous live and asynchronous 
anytime virtual activities, take-home kits, and in-person instruction). While in-person activities 
were slowly reintegrated on a campus-by-campus basis in elementary schools, at-home kits 
allowed students to learn from home with hands-on activities away from the screen. In middle 
schools, ACE Austin worked to target the students most in need of academic support, by 
supplementing tutoring and credit recovery programs in collaboration with the campus teachers, 
counselors, and leaders. All these recruitment strategies helped to increase program 
participation, although numbers remained lower than in previous years. Additionally, the virtual 
format allowed ACE Austin to service more students, especially in asynchronous anytime 
activities that were not restricted to the number of students who could attend the program. All 
centers opened the program to the entire campus. Some campuses also targeted students for 
particular activities, using student-level data, including recommendations from parents, 
teachers, and campus leaders. To help facilitate program recruitment and retention, the DRE 
team created an electronic registration form with accessible, user-friendly dashboards in 
Qualtrics. This is also the platform ACE Austin uses to store and share data that inform program 


improvement, such as student-level academic and attendance records, survey results, and point- 


21 


of-service observations. By having all this information in one place, ACE Austin improved its 


ability to track students’ retention and monitor progress. 


2. Challenge: New learning activities to meet the students’ unique 
needs had to be developed, built on campus assets. 

At the beginning of the year, the implementation of virtual activities faced a steep learning 
curve. Many systems for program implementation had to be significantly redesigned. This was 
especially challenging for program staff because the COVID-19 pandemic led to fluctuations in 
policies and expectations at the district and state levels. As ACE Austin began reintegrating in- 
person activities, ACE Austin staff at some campuses were hesitant to teach in-person classes 
due to safety concerns. Additionally, classes that were held virtually faced the challenge of 
sustaining students’ focus in a remote learning environment. The need to keep students actively 
engaged and participating intensified during remote learning. 

Opportunity: ACE Austin creatively restructured the program processes 
to foster students’ engagement in an online learning environment. 

The COVID-19 pandemic provided program staff with opportunities to revisit the program’s 
structure and processes to meet the program’s goals and expectations. For example, ACE Austin 
site coordinators, family engagement specialists, and project directors attended training in best 
practices for implementing virtual programming and following COVID-19 safety measures, 
offered by AISD, Andy Roddick Foundation, Learn All The Time, and Texas ACE. The project 
directors created two working program guideline documents—one for paid vendors and 
community partners, and one for site coordinators and family engagement specialists. The 
program guidelines clearly outlined expectations and resources and were regularly updated and 
revised. ACE Austin required all ACE Austin frontline staff to complete a program orientation 
each term, at which time, information about the training and the program guidelines were 
disseminated to ACE Austin frontline staff by the site coordinators. Finally, ACE Austin and DRE 
staff modified the ACE Austin Quality Observation Checklist to monitor virtual activities, and 
site coordinators regularly conducted observations of frontline staff to be able to provide 


coaching and one-on-one support, as needed. 


Relying on campus assets and partners, program staff afforded them the opportunity to 
collaborate with each other and creatively redesigned existing and/or new learning activities that 
foster students’ engagement in a digital platform. Finding a space and dedicated time to share 
ideas, expertise, and experiences helped program staff to discuss and explore innovative tools 
and strategies to motivate and engage students in an online environment. When students are not 
engaged, boredom, stress, and anxiety are more likely to manifest. This ultimately results to low 


program participation and attendance. Knowing what tools and strategies to use, and how and 
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when to use them to engage students in newly redesigned online activities, made learning more 


meaningful and interesting. 


3. Challenge: The COVID-19 pandemic created both a public health crisis 
and an economic crisis that necessitated ACE Austin support students 
and families with an unprecedented range of direct resource 
assistance. 


ACE Austin exists within the schools and communities. Due to the economic and health crises, 
ACE Austin served as a community resource and information hub (e.g., serving as a meal 
distribution site or delivering meals; offering mental health services to cope with trauma and 
stress; providing care to children of working families; distributing other resources, such as 
technology; and offering the latest health updates). While this undertaking was laudable, this 
approach expanded the scope of responsibility for program staff and resulted in additional social 
and emotional stress for some program staff, who were balancing taking care of the program 
while concurrently attending to their own personal and family needs. 

Opportunity: ACE Austin created an opportunity to deepen its 
community relations built on trust and partnerships. 

This crisis undoubtedly created opportunities to develop stronger and more long-lasting 
community alliances and partnerships to support the community with a range of direct services. 
Building a network of support systems helped ACE Austin and its staff to support each other. 
With the community at the epicenter, ACE Austin not only legitimized its unique niche to assist 
students with academic needs, but more importantly, also was recognized as a critical agent and 
community partner to support social services and effect positive change. There is no doubt that 
ACE Austin created an opportunity to strengthen its program, which is built on community trust 


that promotes program relevance and sustainability. 
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SUMMARY 

Amidst the COVID-19 pandemic, the ACE Austin Cycle 9 program remained committed to 
providing quality programming that was accessible, flexible, and supportive of the development 
of students’ full potential. The ACE Austin Cycle 9 program offered a variety of high-quality 
activities and services that were designed to support students and families who experienced a 
negative impact from the COVID-19 pandemic. Overall, ACE Austin provided programs and 


services that supported the holistic development of students they served. 


Key Accomplishments 

The ACE Austin Cycle 9 program is aligned with the campus needs assessments and goals 
identified in the campus improvement plans (CIP) of each center. The findings of this report 
were mixed. For example, regular participants who were targeted for math improvement had a 
greater average percentage grade change than did nonregular participants at the elementary 
level only. A greater percentage of regular participants than of nonregular participants who were 
targeted for improvement in school-day attendance improved. Almost all ACE program staff 
agreed the ACE program helped students perform better in academic outcomes, attendance, 
behavior, college/career readiness, and SEL. Additionally, the majority of the program staff 
indicated a desire to continue synchronous live virtual activities on Zoom and asynchronous 
anytime activities in the future. Many of the program staff selected the following three 
professional development areas: aligning with school-day academic goals, developing SEL, and 
engaging with families. Finally, the majority of the teachers at ACE Austin Cycle 9 schools 
agreed that the ACE program benefited the students and families. Despite challenges amidst the 
COVID-19 pandemic, the ACE Austin Cycle 9 program clearly remained committed to providing 
quality programming that was accessible, flexible, and supportive of the development of 
students’ full potential. Table 4 summarizes the major key accomplishments, based on Texas 21% 


CCLC ACE component areas. 
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Table 4. 
Summary of Key Accomplishments 


Program measure and outcome Result 
Student population served © 
Program quality © 
Academics 
Reading © 
Math © 
Grade average © 
School-day attendance © 
Discipline : 
Family engagement 
Program impact 
ACE staff perceptions © 


Teachers’ perceptions © 


Note. Independent sample tests were conducted to compare program participants on each student 
outcome (i.e., reading and math grades, average grades, and school-day attendance rate) between 
regular participants and nonregular participants. 
© Indicates a positive outcome for the measure 
© Indicates a neutral or no change for the measure 
© Indicates a negative outcome for the measure 
Indicates limited or no data available 


Areas for Improvement 

The ACE Austin Cycle 9 program staff remained committed to offering quality programming 
amidst the COVID-19 pandemic. As we strive to go back to normalcy, the ACE Austin Cycle 9 
program staff continue to identify opportunities to assist students in maximizing the benefits of 
program participation. While we achieved a positive impact on many student outcomes this year, 
program managers, site coordinators, and program staff should continue to examine best 
practices to recruit and retain students in the program. Working collaboratively with parents, 
school-day campus teachers, and administrators is key to ensuring that students who need 
assistance are identified and are recruited into the program to take advantage of this free-of- 
charge, quality afterschool programming. Professional development opportunities to improve 


program effectiveness should be provided to ACE staff in areas deemed important. 
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APPENDICES 


Appendix A: Campus-Level Participation 


Table A.1. 
ACE Austin Cycle 9 Campus-Level Participation 
School School Number of poe f naeeiEr famberol 
enrollment _ participants participants participants days of 
participation 
Langford Elementary School 558 194 55 139 30 
Oak Springs Elementary School 251 135 133 2 88 
Rodriguez Elementary School 420 354 41 313 17 
T. A. Brown Elementary School 525 302 71 231 23 
Widen Elementary School 393 150 34 116 24 
Wooten Elementary School 428 141 132 9 68 
Bedichek Middle School 849 348 34 314 11 
Dobie Middle School 739 193 99 94 29 
Martin Middle School 559 114 114 0 133 
Mendez Middle School 649 121 40 81 38 
ACE Austin Cycle 9 5,371 2,052 1,025 1,027 46 


Source. 2020-2021 AISD student records; 2020-2021 ACE data file 
Note. Regular participants are those who participated in the ACE Austin program at least 45 days; nonregular 
participants are those who participated in the ACE Austin program fewer than 45 days. 
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Table A.2. 


ACE Austin Cycle 9 Campus-Level Participants’ Demographics 
Emerging 


School Female Low SES bilingual At risk 
Langford Elementary School (/7= 194) 50% 95% 64% 77% 
Oak Springs Elementary School (7 = 135) 53% 96% 13% 40% 
Rodriguez Elementary School (17 = 354) 48% 94% 55% 64% 
T. A. Brown Elementary School (7 = 302) 50% 91% 74% 78% 
Widen Elementary School (= 150) 49% 93% 45% 61% 
Wooten Elementary School (77 = 141) 50% 88% 74% 80% 
Bedichek Middle School (7 = 348) 49% 83% 35% 73% 
Dobie Middle School (/7= 193) 43% 97% 63% 78% 
Martin Middle School (= 114) 43% 90% 25% 73% 
Mendez Middle School (= 121) 50% 94% 41% 83% 
ACE Austin Cycle 9 (W= 2,052) 49% 92% 51% 71% 
Source. 2020-2021 AISD student records; 2020-2021 ACE data file 
Table A.3. 
ACE Austin Cycle 9 Campus-Level Participants’ Grade Level: Elementary 
School Pre-K Kindergarten 1 2 3 4 5 
Lali Grore Elemeribely S190! 6% 13% 15% 26% 22% 16% 
(n= 194) 
Dako prinies EIeITiENtary 001 15% 15% 11% 12% 13% 13% 16% 
(n= 135) 
een ee 16% 10% 14% 12% 14% 17% 17% 
(n= 354) 
Ty ReBrown Elementary Sciy) 5% 16% 18% 16% 17% 12% 17% 
(n= 302) 
pease ill a 9& 15% 18% 16% 11% 17% 14% 
(n= 150) 
Wooten Eemenitaty Snot) 5% 13% 18% 24% 17% 13% 9% 
(n= 141) 
ACE Austin Cycle 9(W= 1,276) 6% 8% 10% 9% 10% 10% 10% 


Source. 2020-2021 AISD student records; 2020-2021 ACE data file 
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Table A.4. 
ACE Austin Cycle 9 Campus-Level Participants’ Grade Level: Secondary 


School 6 7 8 
Bedichek Middle School (7 = 348) 28% 39% 33% 
Dobie Middle School (7 = 193) 12% 32% 56% 
Martin Middle School (7 = 114) 32% 28% 39% 
Mendez Middle School (7 = 121) 27% 35% 38% 
ACE Austin Cycle 9 (W= 776) 9% 13% 15% 


Source. 2020-2021 AISD student records; 2020-2021 ACE data file 


Table A.5. 
ACE Austin Cycle 9 Campus-Level Participants’ Race 
soni Two or more 
School Asian African Hispanic White 
. race 
American 
Langford Elementary School 8% 39% 2% 1% 
(m= 194) 
Oak Springs Elementary School 34% 65% < 1% < 1% 
(”= 135) 
Rodriguez Elementary School <1% 7% 90% <1% 1% 
(= 354) 
T. A. Brown Elementary School 1% 5% 81% 1% 5% 
(= 302) 
Widen Elementary School 13% 83% 2% 2% 
(m= 150) 
Wooten Elementary School 6% 85% 1% 7% 
(m= 141) 
Bedichek Middle School < 1% 6% 87% 2% 6% 
(7 = 348) 
Dobie Middle School 
{1} t\) 0, t1) 

(n= 193) 3% 8% 85% 5% 
Martin Middle School <1% 23% 69% 6% 
(m= 114) 
Mendez Middle School < 1% 12% 87% 
(m= 121) 

ACE Austin Cycle 9 (W=2,052) 2% 10% 84% 1% 3% 


Source. 2020-2021 AISD student records; 2020-2021 ACE data file 
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Appendix B: 


Program Quality Observation Domains and Scales by Activity Format 


Asynchronous 
Synchronous : 
In-person ee anytime 
live virtual : 
virtual 
Physical Safety 
Students enter the classroom quickly and safely. X 
The space is free of health and safety hazards and suitable for the activity. X 
Only staff open doors to program spaces (includes outside and inside , 
doors). 
A system for monitoring bath breaks is in place. X 
All students are signed into the ACE attendance sheet. X 
The instructor's virtual space is classroom appropriate and professional. X X 
No photos, videos, recordings, or screenshots are being taken during the 7 : , 
duration of the activity without consent. 
Instructions include necessary precautions (students know when to get ; P 
parent involved; e.g., using scissors for pre-K). 
Staff are not leading a 1-on-1 activity with a student (unless another adult ? - 
is present). 
*Overall score for physical safety (global rating) 1 to 5 scale X X Xx 
Emotional Safe 
The climate is positive, encouraging and mutually respectful. X X X 
Staff communicate to students respectfully. Staff never belittle, talk down : : ° 
to, show bias toward, or yell at the students. 
Staff ask about and acknowledges feelings of students X X 
Staff make time to talk with students individually and does not single out ‘ 
students in front of their peers. 
When managing feelings, staff help students to respond appropriately and ‘; y 
allows them time to take a break if necessary. 
When students experience conflict, staff offer solutions for student to : ‘: 
choose from or encourages them to come up with their own solutions 
Staff work side by side with students, participating if appropriate. Staff do : 
not separate themselves to work on other tasks. 
Staff circulate and interact with every student. X X 
External online resources are classroom appropriate and reputable and have : . 
been vetted prior to the activity. 
Staff use SEL strategies when necessary or appropriate (modeling coping y ' 5 
strategies, deep breathing, brain breaks). 
Materials and videos are culturally inclusive. X X X 
*Overall score for emotional safety (global rating) 1 to 5 scale X X X 
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Clear Expectations 
Staff model appropriate behavior through actions and language. 
Staff establish a reasonable classroom routine. 


Students are allowed to take care of practical needs (get supplies, clean up). 


Very little time is spent managing behavior and students respond quickly to 
redirection when given. 


When consequences are given, they are logical and appropriate. 


Staff use attention-getters to quickly gain students’ attention as needed. 


* Overall score for clear expectations (global rating) 1 to 5 scale 


/ntroduction 


All materials and supplies are ready at the beginning of the session, and 
there are enough supplies for everyone. 


Activity requires no supplies or minimal household supplies. 


Activity requires no supplies or minimal household supplies, unless supply 
kits have been provided in advance. 

Staff capture students’ attention with an active icebreaker preferably 
related to the learning or skill building objective. 


Staff make connections to prior knowledge. 


Staff clearly state and post a specific learning or skill-building focus for the 
session. 


*Overall score for introduction (global rating) 1 to 5 scale 


Intentional Skill-huilding Activity 
Staff begin by briefly demonstrating the skill students will practice. 


For the majority of activity time, staff support students in guided skill 
practice, assisting as necessary. 

Staff end the hands-on activity by stepping back and allowing students to 
practice the skill with minimal or no assistance while informally assessing 
their progress toward the learning objective. 

Throughout the activity, staff provide structured opportunities to talk about 
what they are learning by asking open-ended guiding questions. 


Throughout the activity, staff break difficult tasks into smaller steps. 


Throughout the activity, staff allow “wait time” for students to solve 
problems on their own. 


* Overall score for intentional skill building (global rating) 1 to 5 scale 


In-person 


Synchronous 
live virtual 
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Asynchronous 
anytime 
virtual 


Synchronous 
In-person dea ack 
live virtual 

Reflection 
Staff reiterate the learning or skill-building objective. Xx X 
Staff allow structured time for students to reflect on the activity and assess : ’ 
the progress toward the learning or skill building objective. 
Staff acknowledge accomplishments and challenges. Xx X 
*Overall score for reflection (global rating) 1 to 5 scale X X 


Note: This table contains the quality program observation domains and the scaled items used by program 
director and site coordinators to observe the various program activities, based on the YPQA (Smith, et.al., 2016). 
The ‘x’s denote the items that apply for each of the domains of the program quality observations conducted in 
different delivery formats (i.e., in-person, synchronous live virtual, and asynchronous anytime virtual), using a 5- 
point Likert scale. 
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Asynchronous 
anytime 
virtual 
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